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January 13, 2009 

MEMORANDUM FOR UNDER SECRETARY OF DEFENSE FOR INTELLIGENCE 
UNDER SECRETARY OF DEFENSE FOR ACQUISITION, 
TECHNOLOGY & LOGISTICS 

SUBJECT: Final Report of the Defense Science Board (DSB) Advisory Group on 
Defense Intelligence Task Force on Operations Research Applications for 
Intelligence, Surveillance and Reconnaissance (ISR) 

I am pleased to forward the final report of the DSB Advisory Group on Defense 
Intelligence Task Force on Operations Research Applications for ISR co-chaired by Dr. 
Robert Lucky and Major General Richard O’Lear, USAF (Ret). This Task Force was 
sponsored by the Under Secretary of Defense for Intelligence (USD(I)). 

As requested in the Terms of Reference (TOR) the Task Force was asked to assess 
the use of operations research (OR) methods within the DoD, determine how OR 
supports DoD decision making for ISR programs and recommend needed improvements. 

The final report addresses the taskings in the TOR and provides two test cases for 
using OR methodologies. The report provides recommendations for the USD(I) which 
include establishing an organic ISR OR oversight function within USD(I) as well as a 
capability to conduct OR for strategic investment decisions. Further recommendations 
address the details of the OUSD(I) OR function. 

I endorse the Task Force’s findings and recommendations and encourage you to 
review the report. 



Dr. William Schneider, Jr. 
DSB Chairman 
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MEMORANDUM FOR THE CHAIRMAN, DEFENSE SCIENCE BOARD 


SUBJECT: Final Report of the Defense Science Board (DSB) Advisory Group on Defense 

Intelligence Task Force on Operations Research Applications for Intelligence, Surveillance and 
Reconnaissance (ISR) 


Attached is the final report of the DSB Advisory Group on Intelligence Task Force on 
Operations Research Applications for ISR. The Task Force was asked to examine the use of 
operations research (OR) methods to support ISR decision making within the Department of 
Defense (DoD). 

The Terms of Reference asked the Task Force to survey the departments and agencies to 
determine how much OR is being performed; assess how the results of OR are being used in 
decision making; recommend a test case(s) for using OR methodologies; and recommend steps 
the Department can take to institutionalize the use of OR in future DoD decision making. 

The Task Force reviewed the application of OR - which it views as decision support analysis 
based on information collected and analyzed using the scientific method - throughout the DoD 
and the Services focusing primarily on the resources available to conduct OR, as opposed to 
reviewing specific programs. The Task Force examined the commitment of decision makers to 
the use of OR and the extent to which these resources reflected the application of standard OR 
processes and practices and the ma nn er in which the results appeared to be employed. The Task 
Force also looked at several private-sector OR application models and at recent and historical 
uses of OR in support of national security requirements. 

Based on its inquiry the Task Force arrived at the following observations: 

• Operations Research represents a powerful tool to help improve the quality of 
investment decision making by illuminating key issues, assumptions, and sources of 
information. 

• Operations Research is applied inconsistently throughout the Defense and ISR 
co mm unities and each lacks standard OR processes and practices, and consistent 
organizational models or co mm itments. 


• OR and the use of it can be strengthened in the Defense and ISR communities 
through effective institutionalization. 

• The utility of OR can be more firmly established through appropriate test cases. 

Based on its findings, the Task Force recommends establishing an organic OR oversight function 
and investment decision capability within the USD(I). In response to the TOR, the Task Force 
recommends two test cases, one related to the balance in the intelligence cycle and the other 
related to investment decision making in biometrics technologies. 

The attached report provides the rationale for the Task Force’s findings and recommendations, 
reflects on the historical use of OR, details current OR practices, specifies attributes required for 
the test cases, and offers potential institutionalization models. 

We appreciate the contributions made by the Department of Defense, Intelligence Community, 
and private sector professionals who took the time to provide us with their knowledge and 
expertise; the members of this study; the Executive Secretary; and the DSB Secretariat and its 
military assistant. 




Co-Chairman 


Co-Chairman 
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Executive Summary 

The Advisory Group on Defense Intelligence (the Task Force) of the Defense Science 
Board (DSB) was tasked by the Under Secretary of Defense for Intelligence (USD(I)) to 
examine the manner and extent to which Operations Research (OR) is employed by the 
Department of Defense; how OR can be used to support Intelligence, Surveillance, and 
Reconnaissance (ISR) decision making; and the manner in which OR can be 
institutionalized in the Department of Defense (DoD). 

The Task Force reviewed the application of OR 1 - which it views as decision support 
analysis based on information collected and analyzed using the scientific method - 
throughout the DoD and the Services. The review was done on a selective basis, 
focusing principally on the resources available to conduct OR as opposed to specific 
programs. The Task Force looked at decision makers' commitment to the use of OR, the 
extent to which resources reflected the application of standard processes and practices, 
and the manner in which the results of resources appeared to be employed by the 
organizations of which they are a part. The Task Force also looked at several private- 
sector OR application models (notably FedEx), and at recent and historical use of OR in 
support of national security requirements. 

Four aspects of our inquiry stand out: 


1. Operations Research represents a powerful tool to help improve the quality 
of investment decision making by illuminating key issues, assumptions, and 
sources of information. 

2. Operations Research is applied inconsistently throughout the Defense and 
ISR communities. These communities do not possess standard OR processes 
and practices, a consistent organizational model, or a consistent 
commitment to the use of OR. 

3. OR - and its use - can be strengthened in the Defense and ISR communities 
through effective institutionalization. The Task Force commends to the 
USD(I) for further consideration models employed by the private sector (of 
which FedEx appears to be a strong examples) and the Army (TRAC and CAA). 
The Military Operations Research Society (MORS) is a domain expert 
resource that should be used in building a plan for institutionalization. 

4. The utility of OR can be more firmly established through appropriate test 
cases. The Task Force points to two cases for USD(I) consideration: 
Biometrics and Investment in a Balanced Intelligence Cycle, with specific 
emphasis on Unmanned Aerial Systems (UASs). 


1 The analytical study of military problems undertaken to provide responsible commanders and staff agencies with a scientific basis for 
decision on action to improve military operations. Also called operational research; operations analysis." (JCS Pub 1-02) 
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The Task Force believes Operations Research can strengthen the quality of ISR decision 
making in terms of the ability of investment decisions to more clearly reflect 
requirements, alternatives, resources, and technologies. However, OR is currently 
employed throughout DoD and the Services on an inconsistent basis, reflecting varying 
levels of leadership commitment and support, a lack of standards regarding OR training 
and methodologies, and uncertain career recognition and prospects for OR 
practitioners. Although the Federal Acquisition Regulations (FAR) require the use of OR 
in support of Major Acquisition Programs, OR is used inconsistently in ISR investment 
decision making, even on major systems, and the resources employed do not reflect 
standard practice. In general, OR is used to configure existing resources or to support 
(or refute the need for) additional "buys" of existing systems. OR is also used to support 
complex mission management decisions regarding operational systems. Overall, 
however, the use of Operations Research is diminished by a seeming lack of leadership 
support for its application in support of program investment, career investment, and 
sufficient resource investment to create sustained excellence in its practice throughout 
the ISR and larger DoD communities. In contrast, OR (and science method-based 
analysis) were disciplines prized for their ability to help decision makers meet significant 
challenges, particularly in support of U.S. military operations during World War II, in 
determining force mix and sufficiency after WWII, and in rationally mobilizing the U.S. 
industrial base in support of national security requirements. While there exist islands of 
OR practice today - some of them quite competent - in DoD and the Services, 
institutional capacity to apply science-based analysis to complex decisions is far less 
than it has been in the past, and less than it should be. 

Institutionalization of OR for ISR investment decision making requires a sound 
management model and the sustained support of DoD leaders, both in its application 
and sustainment of the resources necessary to keep OR a robust discipline throughout 
the Department and the Services. Such commitment will be required to reach a level of 
competent OR practice throughout the ISR community. At present, competent OR cells 
exist, but are poorly connected as a professional community and inconsistently utilized 
by decision makers. The Task Force believes OR should be institutionalized, practiced 
consistently (from a standards and methodology perspective), and used routinely to 
support high value decisions involving complex requirements and significant resources. 
The Task Force understands clearly that OR does not "make" decisions; rather OR can 
and should make visible the sources of data and information used to make decisions, 
help drive the development of consistent assumptions and requirements, and give an 
objective basis against which decisions can be weighed. The Task Force recognizes that 
complex decisions exist within a wider context of policy and changing requirements; OR 
provides an objective "anchor" that decision makers should consider. 

The Task Force recommends that the Federal Express (FedEx) management model be 
given strong USD(I) consideration as a sound example of the use of OR in the private 
sector. In addition, the work being done by the Army's Training and Doctrine 
Command's (TRADOC) Analyses Center (TRAC) and the Center for Army Analysis (CAA) 
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provide good examples of utilizing OR within the government. The private sector model 
used by FedEx employs OR at a corporate level to decide which lines of business are 
competitive and at what level; employs OR at the line-of-business level for investment 
decision making and configuration of operational resources; and provides corporate 
oversight to ensure the presence of consistent OR standards methodology throughout 
the company's family of businesses. FedEx does not "require" the use of OR instead the 
FedEx leadership commitment to OR-based decision making creates a culture which 
gives its practitioners substantial influence during high value decision making. At the 
same time, the Task Force believes that TRAC may represent a strong example of an 
embedded, "line-of-business" capability, analogous to the OR capabilities in FedEx's 
various lines-of-business. Overall, the Task Force recommends the USD(I) establish an 
organic 2 ISR OR oversight function that requires, sustains, and reviews OR standard 
processes and practices in DoD ISR investment decisions. 

In response to the TOR, the Task Force recommends two ISR test cases. The first related 
to investment decision making in biometrics technologies in support of ISR 
requirements; and the second to investment decision making for balance in the overall 
intelligence cycle, and specifically in regard to UASs. These test cases should be 
structured to illuminate challenges and opportunities to strengthen OR decision making 
in the ISR investment domain. The biometrics test case provides an opportunity to help 
biometrics meet its ISR potential through stronger investment strategy decision making, 
creation of a real biometrics information architecture, and development of a biometrics 
information sharing environment useful to both intelligence analysts and warfighters. 

Building a more effective OR capability to support ISR is a difficult but meaningful 
challenge. The Task Force believes that the Military Operations Research Society 
(MORS) is a resource the USD(I) can use to help implement this study's 
recommendations. 

In summary, the decision maker must commit to using the OR results in support of 
decisions to be made. While it is not realistic to believe that an important ISR 
investment decision will be driven solely by OR analytical results, it should be clear to 
stake-holders that the results were carefully considered in the decision-making process. 

The Task Force looks forward to discussing with the USD(I) and the ISR community its 
recommendations and is grateful for this opportunity to serve the larger national 
security community. 


2 An in-house OR function. 
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Chapter 1: Introduction and Tasking 

As a result of the recent High Value Targets (HVT) Study performed by Edge Consulting, 
Inc. to examine the application of ISR assets in support of operations in the U.S. Central 
Command's Area of Responsibility (USCENTCOM AOR), senior leaders in the Office of 
the Under Secretary of Defense for Intelligence (OUSD(I)), Program Analysis and 
Evaluation (PA&E), and the Joint Staff engaged in a discussion about the merits of 
operations research in support of planning, programming, and operations. 3 This 
discussion led to a request by the USD(I) to the Defense Science Board (DSB) to perform 
a study on the use of Operations Research methods to support decision making within 
the Department, with a focus on Intelligence, Surveillance, and Reconnaissance (ISR) 
investment choices. The DSB assigned this task to its Advisory Group on Defense 
Intelligence (the Task Force). The Terms of Reference (TOR) directed the Task Force to 
examine the manner and extent to which OR is employed by the Department of Defense 
(DoD), how OR can be used to support ISR decision making, and the manner in which OR 
can be institutionalized in the DoD. The TOR included the following four tasks: 

1. Survey the departments and agencies to determine how much OR is being 
performed; 

2. Assess how the results of that OR are being used in decision making; 

3. Based on these findings, recommend a test case for using OR methodologies 
to improve DoD ISR investment decisions; and 

4. Recommend steps the Department can take to "institutionalize" the use of 
OR in DoD decision making in the future. 

The Task Force discussed the scope of the OR inquiry that would best serve the USD(l)'s 
needs and considered limiting the inquiry to an examination of OR as a component of 
technology and acquisition strategy. The Task Force also considered extending the 
inquiry to ways in which OR can be used to strengthen operational planning. In the end, 
the Task Force concluded that a broad inquiry was appropriate to the TOR tasks 
provided by the USD(I). Such an inquiry included examination and discussion of OR as a 
component of operational planning, as an element in resource allocation in support of 
operations, and as an element of acquisition and technology strategy pertinent to 
supporting resource allocation and operational planning. The Task Force believes this 
broad-based approach is particularly relevant to the ISR challenges facing the 
Department now and in the foreseeable future. Secretary of Defense Robert Gates 


3 The Office of the Undersecretary of Defense (Intelligence), Program Analysis and Evaluation, and the Joint Staff co-sponsored Edge 
Consulting, Inc. to deliver a series of quantitative ISR performance analyses. The Department's objective was to enrich the 
standard methods - interviews, surveys, war games - with quantitative analysis of empirical operations and intelligence performance data. 
As one example, the Department tasked Edge with quantifying the contribution of a dozen Intelligence Surveillance and Reconnaissance 
(ISR) systems to the special operations campaign against High Value Targets (HVT). Working in close cooperation with the Department, 
U.S. Central Command, and Special Operations Forces, Edge gathered over one million empirical records associated with thousands of 
HVT operations. Operations Research techniques were adapted and applied to quantify the contribution of ISR systems to successful 
operations. Causal factor analysis was applied to understand root causes of poor performance. The Department and special operations 
forces applied these results to adjust both acquisition and operational decisions. 
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described some of these challenges - related to requirements of ever increasing 
complexity and constrained resources - in a speech on September 29, 2008 at the 
National Defense University. 4 His speech highlighted the challenges associated with 
employing complex platforms and capabilities against an ever-changing set of missions 
and requirements. These challenges apply clearly to current and prospective ISR 
investments. 

Given this complex environment, the Task Force believes that OR can strengthen end- 
to-end analysis associated with coupling long-term investment strategies, requirements, 
operational concepts, and operational plans. The Task Force received 22 briefings and 
questioned representatives of the Services, various operational agencies, planning and 
evaluation specialists from PA&E, and program management cadres within the DoD and 
Office of the Director of National Intelligence (ODNI). The Task Force also spoke with 
industry representatives who provided useful insights regarding ways in which OR can 
be used for strategic investment planning and in support of missions associated with 
specific lines of business. Appendix C lists the organizations that briefed Task Force. 

The Task Force believes its broad approach to the TOR tasking provides the USD(I) with 
insights pertinent to a full range of ISR investment needs, ranging from long-range 
technology strategy to the efficient use of existing ISR resources. In regard to 
methodology, the Task Force focused principally on the resources available to conduct 
OR, instead of reviewing specific programs, a task beyond the means and time available 
for this effort. The Task Force looked at the extent to which these resources reflected 
the application of standards and the manner in which the results appeared to be 
employed by the respective organizations. This approach allowed the Task Force to 
infer a lack of OR use and standard processes and practices in ISR investment decision 
making. Inquiries into specific programs and investments could more precisely establish 
the manner and competence with which OR has been or is being employed. 


4 As delivered by Secretary of Defense Robert M. Gates, Washington, D.C., Monday, September 29, 2008. See: 
http://www.defenselink.mil/speeches/speech.aspx?speechid=1279 
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Chapter 2: The Task Force's Understanding of Operations Research 


Joint Chiefs of Staff Publication 1-02 (JCS Pub 1-02), Department of Defense Dictionary of 
Military and Associated Terms, defines Operations Research as "The analytical study of 
military problems undertaken to provide responsible commanders and staff agencies 
with a scientific basis for decision on action to improve military operations." Operations 
Research may also be referred to as Operational Research or Operations Analysis. 


The Task Force received numerous briefings from Operations Research (OR) 


OR CAN SIGNIFICANTLY 
INCREASE THE DECISION 
MAKER'S INSIGHT AND 
IMPROVE THE QUALITY OF 
DEBATE AROUND 
KEY DECISION ISSUES. 


practitioners from a variety of organizations. Information 
gleaned from these briefings convinced the Task Force that 
the application of structured decision making tools and 
processes can provide significant assistance to the decision 
maker in the analysis of complex operational and investment 
decisions. OR can increase significantly the decision maker's 
insight and also improve the quality of the debate around 
the key decision issues. 


As noted in Chapter 1, the Task Force's understanding of OR 
- decision support analysis based on information collected and analyzed using the 
scientific method - broadly encompasses analysis of operations as well as strategic 
technology, investment, and acquisition planning in support of operations. OR 
practitioners have a variety of tools at their disposal. These tools consist of applied 
mathematical techniques such as statistical and data analysis, linear and non-linear 
regression, queuing theory, stochastics, as well as linear and non-linear programming. 
Additionally, practitioners have access to a variety of off-the-shelf models and 
simulations, as well as commercial software packages. For those problems where off- 
the-shelf tools are not practicable, OR practitioners can develop models and simulations 
using the techniques listed above. 


It is the Task Force's opinion that OR should be used as an important component of an 
analytical approach to decision making. Information provided to the Task Force by 
briefers also assisted in formulating a picture of what good OR practices should entail. 

2.1 Desired Attributes of Operations Research 

The Task Force viewed OR as a scientific method; a method based on gathering 
empirical and measurable evidence or data and subjecting it to repeatable principles of 
reasoning. These principles of reasoning often involve the formulation and testing of 
hypotheses. This process requires a thorough identification of the problem to be 
solved, which includes advance agreement by decision makers on the objective 
function, 5 metrics, data provenance, and the study assumptions and boundary 


5 The Task Force considered at length the manner in which valid objective functions could be employed in the OR analysis of ISR 
investments. We considered established "attrition models" by which the effectiveness of a system is assessed as a function of the extent 
to which it contributes to the attrition of adversary forces, as well as the extent to which an ISR system can reduce the level of uncertainty 
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conditions - all of which are keys to solid analysis. The best use of OR occurs when 
employed by a team composed of operators and analysts capable of investigating the 
entire problem and conducting analyses without bias. In so doing, a good OR team will 
seek agreement on assumptions and objective functions, and advocacy from all key 
stakeholders up-front, and continue to cultivate that advocacy throughout the analytic 
process. 

2.2 Advantages of Operations Research 

Good analysis involves all decision makers and stakeholders from the very beginning. 
This involvement includes, but is not limited to, agreement on the actual questions to be 
answered by the analysis as well as all assumptions and constraints pertinent to the 
decision to be made. Such involvement by senior leadership facilitates periodic reviews 
and assessments of progress, and provides the opportunity to redirect or refine 
guidance to the study team if needed. A repeatable process based on the scientific 
method enhances the likelihood that the analysis will be objective and the results will 
provide the organizational leadership with the analytic underpinnings to make sound, 
informed decisions. Structured analyses will provide decision makers with clear 
traceability from study requirements to final results. In addition, analyses can often 
illuminate problems previously unidentified or unknown to senior leadership. 

2.3 ISR Decisions to which Operations Research is Applicable 

OR readily lends itself to assisting organizational leadership in making high-value 
decisions in which significant consequences and resources are at stake. Such decisions 
involve risk and uncertainty, and are often driven by complex requirements involving 
multiple stakeholders with competing objectives. These analytic processes can be used 
across all ISR disciplines focused on a prioritized list of current and future threats and 
missions. Tailored approaches can be applied to the entire intelligence cycle including 
system requirements analysis, asset tasking, data collection, processing and 
exploitation, as well as intelligence analysis. 

2.4 Conditions under which Operations Research can be Institutionalized 

OR cannot be institutionalized without strong commitment, support, and advocacy from 
senior leadership, which is assumed to have the authority to implement decisions 
influenced by the OR process. The senior leadership to which the Task Force is referring 
includes those individuals responsible for the outcome of the decision process, that is, 
those that will make the decision based upon various inputs. 

This approach requires senior leadership to assemble and sustain a critical mass of OR 
resources to include a well-trained and adequately-funded staff. OR professionals need 
career development and prospects. The Task Force found noteworthy the approach 


regarding a question of intelligence interest. The Task Force recognizes that establishing objective functions will continue to challenge 
decision makers, although we believe good work has been done to establish Key Performance Parameters (KPPs) for many ISR systems. 
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employed by the Army (TRAC) in building and sustaining a cadre of competent OR 
professionals. 

The essential requirement for effective OR in support of decision making is the support, 
confidence, and early participation of the decision maker. The decision maker must 
ensure that the question asked, the assumptions made, the objective function defined, 
and the metrics chosen all meet the needs of the decision process. 

A decision maker who demands good analytical support for decisions will get it. 
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Chapter 3: Operations Research - Historical Context 

The Task Force examined, at a high level, OR from a historical perspective. The Task 
Force looked at OR in the context of its development and use principally in the U.S., with 
an understanding that the OR discipline is not confined to the U.S. This review of OR's 
historical context (from World War I to present day requirements and acquisition 
processes) provided the Task Force with useful insights regarding the need for 
standards, involvement of operators alongside analysts, and senior leadership 
commitment. 

World War I 

During World War I civilian scientists on both sides of the Atlantic joined together to 
apply scientific methods to defeating the forces of the Triple Alliance, specifically the 
submarine threat. In 1915, after reading one of Thomas Edison's speeches on military 
preparedness published in The New York Times, Secretary of the Navy Josephus Daniels 
came upon the idea of the Naval Consulting Board. After discussions with Edison, it was 
determined that a board consisting of the brightest scientific and industrial minds in the 
country would be convened under Edison's chairmanship. Its primary purpose was to 
advise the Navy on systems acquisition. Edison, fully aware of the threat the U-boat 
posed to merchant shipping and Britain's survival, focused his talents on developing 
systems and tactics to be used in the fledgling anti-submarine warfare (ASW) effort. He 
employed many applied mathematics techniques, such as time-motion analysis to 
understand the effects of zigzagging in disrupting fire control solutions; graphical and 
statistical data analysis to understand U-boat operations; threat system assessments; 
and military wargaming. 6 That same year the British Admiralty instituted the Board of 
Invention and Research to encourage the naval service to solicit expert technical 
assistance in solving tactical problems. Chaired by Sir Jackie Fisher, former First Sea 
Lord, it was comprised of numerous scientists and divided into six science and 
technology sections. Of these, Section II: Submarines and Wireless Telegraphy, was the 
most important. In the two years of the Board's existence, its members evaluated over 
41,000 submitted inventions. 7 

British use of OR during World War II 

Operations Research (Operational Research in the United Kingdom and Canada) was 
formally established in the late 1930's at a time when Adolf Hitler announced his 
intentions to create a Luftwaffe equal to the combined air forces of Britain and France. 
To defend Britain against an attack from Germany, the British government established 
radar installations along the coast to provide the Royal Air Force (RAF) with early 
warning of incoming German strikes. However, in 1938, tactical exercises demonstrated 
that the system fell short of its operational requirements. A.P. Rowe, Superintendent of 


6 Scott, Lloyd N., Naval Consulting Board of the United States, Government Printing Office, Washington, D.C., 1920. 

7 Hackmann, Willem, Seek & Strike: Sonar, Anti-Submarine Warfare and the Royal Navy 1914-54, Her Majesty's Stationary Office (HMSO), 
London, 1984. 
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the Bawdsey Research Station, proposed that research into operational aspects rather 
than technical aspects of the system should be conducted - thus coining the phrase 
"operational research." 8,9 These studies led to the establishment of formal techniques 
of command and control of fighter versus bomber operations in the Battle of Britain. 

Operational Research spread to other organizations throughout the British military 
establishment. Operational Research sections in the Coastal Command and the 
Admiralty were responsible for developing convoy procedures to keep Britain's lifeline 
to North America open, perfecting the use of airborne radar in detecting U-boats, and 
shutting down the U-boat transit lanes from bases along the Bay of Biscay to open 
ocean operating areas. 10, 11, 12 Other sections were established to support the Army 
(primarily the Antiaircraft Command), as well as RAF Bomber Command where studies 
in navigation accuracy, radar performance, and the employment of chaff contributed to 
improved bombing accuracy and aircraft survivability during the area bombing of 
German cities. 13 Recent studies in the United Kingdom indicate that the area bombing 
campaign undertaken by Bomber Command in 1942, and espoused by the Commander- 
in-Chief, Sir Arthur Harris, may have been validated by the misapplication of operations 
research and was of less impact on the German war economy and civilian morale then 
was believed at the time. 14 

U.S. use of OR during World War II 

American scientists became involved in the war effort in June 1940 with the formation 
of the National Defense Research Committee (NDRC) and the subsequent formation of 
the Office of Scientific Research and Development (OSRD) under Dr. Vannevar Bush. 15, 
16 Like the Naval Consulting Board of World War I, both the NDRC and OSRD were 
comprised of some of the brightest minds from academia and industry. Upon the entry 
of the United States into the war, the Navy took the lead in establishing operations 
research organizations. In March 1942 the Mine Warfare Operations Research Group 
(MWORG) was established at the Naval Ordnance Laboratory. MWORG went through 
numerous reorganizations throughout the war, and ultimately supported the Army Air 
Forces XXI Bomber Command in planning Operation Starvation in the Far East - the 


8 Cunningham, W. Peyton, et al, "Of Radar and Operations Research: An Appreciation of A.P. Rowe (1898-1976)," Operations Research, 
Volume 32, Number 4 (July-August 1984), pp. 958-967. 

9 Larnder, Harold, "The Origin of Operational Research," Operations Research, Volume 32, Number 2 (March - April 1984), pp. 465-475. 

10 Falconer, N., "On the Size of Convoys: An Example of the Methodology of Leading Wartime OR Scientists," Operational Research 
Quarterly (1970-1977), Volume 27, Number 2, Part 1 (1976), pp. 315-327. 

11 McCue, Brian, U-boats in the Bay of Biscay: An Essay in Operations Analysis, National Defense University Press, Washington, DC, 1990. 

12 McCloskey, Joseph F., "British Operational Research in World War II," Operations Research, Volume 35, Number 3 (May - June 1987), 
pp. 453-470. 

13 Ibid. 

14 Kirby, M. and R. Capey, "The Area Bombing of Germany in World War II: An Operational Research Perspective," The Journal of the 
Operational Research Society, Volume 48, Number 7 (July 1997), pp. 661-677. 

15 Meigs, Montgomery C., Slide Rules and Submarines: American Scientists and Subsurface Warfare in World War II, National Defense 
University Press, Washington, DC, 1990. 

“ Fortun, M and S.S. Schweber, "Scientists and the Legacy of World War II: The Case of Operations Research (OR)," Social Studies of 
Science, Volume 23, Number 4 (November 1993), pp. 595-642. 
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mining of Japan's Inland Sea. During the campaign, more than 1,000 ships were 
damaged or sunk, including over 100 naval vessels, greatly reducing Japanese shipping 
in this area. 17,18 

When the U.S. entered WWII, the loss of shipping to U-boats in the Atlantic rapidly 
reached catastrophic proportions. In March 1942, Captain Wilder D. Baker, Commander 
of Destroyer Squadron 31 formed the Anti-Submarine Warfare Unit in Boston. It did not 
take the Unit long to conclude that the major cause of America's poor ASW campaign 
was inadequate tactical doctrine. By the end of March, Baker approached Professor 
Philip M. Morse of MIT and asked him to undertake the role of lead analyst in what 
would be called the ASW Operations Research Group (ASWORG). One of the first issues 
ASWORG tackled was the problem of search, by ship or plane, for submarines and 
warships. Its first memorandum, published 1 May 1942, was titled "Preliminary Report 
on the Submarine Search Problem." The recommendations in this report were 
immediately put into effect and improved the tactics of convoy protection and the 
search for U-boats. As successes grew, ASWORG gradually undertook studies in the 
Pacific, first on submarine operations, then naval air activities, and finally on all aspects 
of operations. 19, 20 During this time Operations Research was successfully employed in 
both the Army and Army Air Forces as well. 21 

OR and the Cold War 

After World War II, acceptance of Operations Research techniques became widespread 
throughout industry leading to the establishment of professional societies in the United 
States, United Kingdom, and Canada. OR's utility to military operations continued with 
the introduction of Federally Funded Research and Development Centers (FFRDCs), first 
with the Center for Naval Analyses, then Project Air Force, and then followed quickly by 
Lincoln Laboratories. 22 At the outset of the Cold War, Operations Research found new 
missions, most notably in the development of the Single Integrated Operational Plan 
(SIOP), a plan and planning process to determine the use of U.S. nuclear weapons in the 
event of nuclear war. The first such plan was submitted in 1960. Operations Research 
tools are still employed to study blast, thermal, and radiation effects, as well as mission- 
asset allocation. 23 However, since 1983 this capability has diminished significantly. 24 


17 Shortley, George, "Operations Research in Wartime Naval Mining," Operations Research, Volume 15, Number 1 (January-February 
1967), pp. 1-10. 

18 McCloskey, Joseph F., "U.S. Operations Research in World War II," Operations Research, Volume 35, Number 6 (November - December 
1987), pp. 910-925. 

19 Tidman, Keith R., The Operations Evaluation Group: A History of Naval Operations Analysis, Naval Institute Press, Annapolis, MD, 1984. 

20 Morse, Philip M., "The Beginnings of Operations Research in the United States," Operations Research, Volume 34, Number 1 (January - 
February 1986), pp. 10-17. 

21 Shrader, Charles R., History of Operations Research in the United States Army, Volume 1:1942-1962, United States Army, Washington, 
DC, 2006. 

22 U.S. Congress, Office of Technology Assessment, A History of the Department of Defense Federally Funded Research and Development 
Centers, OTA-BP-ISS-157, U.S. Government Printing Office, Washington, DC, 1995. 

23 Pringle, Peter and William Arkin, SIOP, the Secret U.S. Plan for Nuclear War, Norton Publishing, New York, New York, 1983. 


Operations Research Applications for ISR 


15 



Chapter 3: Operations Research -Historical Context_ 


Also in the early 1960s, Secretary of Defense Robert McNamara, convinced that the 
Secretary of Defense should control the evaluation of military needs, established the 
Systems Analysis discipline as an administrative tool to estimate the required quantity 
and performance of forces and weapons in relation to their mission and costs known as 
the Planning, Programming and Budgeting System (PPBS). 25 The effort was not entirely 
successful - "McNamara's systems analysis instilled structure and discipline, too, but in 
the Pentagon, it was applied in an atmosphere of competition and distrust. He 
introduced systems analysis explicitly to obtain control over Service programs. The 
mutual trust and regard in the fleet were impossible in Washington. Decision making by 
analysis took on a hostile, jaded, and onerous cast." 26 

Goldwater-Nichols Act 

In 1986, the Goldwater-Nichols Department of Defense Reorganization Act called on the 
Joint Chiefs of Staff to advocate the effectiveness of joint operations. In so doing, it 
tasked the Vice Chairman to chair a special council on military requirements known as 
the Joint Requirements Oversight Council (JROC) to provide programmatic 
recommendations to the Chairman of the Joint Chiefs for consideration and submission 
to the Secretary of Defense. 27 This Act positioned the Joint Staff to be the only military 
body to address force integration and to resolve inter-service conflict with a total 
organizational perspective. It assigned the Chairman of the Joint Chiefs of Staff (CJCS) 
with determining strategy, military requirements, programmatic prioritization, and 
budget recommendations. The fact that the Joint Staff is comprised of military 
personnel avoided the problems encountered with the McNamara PPBS. 28 In 1994, 
Admiral William Owens, Vice Chairman of the JCS established the Joint Warfighting 
Capability Assessment (JWCA) process to provide better advice to the CJCS by increasing 
the involvement of the Combatant Commanders and the Services in the JROC process. 
The JWCA process "examines relationships and interactions among warfighting 
capabilities and identifies opportunities for improving joint effectiveness." 29 Each team 
is comprised of representatives from the Unified Commands, OSD, and the Services and 
is chartered with assessing areas where capabilities gaps or duplication may exist, as 
well as determining where technologies may be brought to bear. The JWCA findings are 
then presented to the JROC for consideration. The process has been instrumental in 
providing a deeper analytic underpinning to the recommendations from the CJCS to the 
Secretary. 


24 Technology Strategies & Alliances, Prepared for the Office of Net Assessment, Maintaining Advantage in a Multi-polar Nuclear World: 
Declining Resources Final Report, December 2007. 

25 Enthoven, Alain and K. Wayne Smith, How Much is Enough?: Shaping the Defense Program 1961-1969, RAND Corporation, Santa 
Monica, CA. 

26 Hughes, Wayne P., "Navy Operations Research," Operations Research, Volume 50, Number 1 (January - February 2002), pp. 103-111. 

27 Owens, William A., "JROC: Harnessing the Revolution in Military Affairs," Joint Forces Quarterly, Summer 1994, pp. 55-57. 

28 Turcotte, Dr. William E., "Service Rivalry Overshadowed," Airpower Journal, Fall 1996, pp. 28-33. 

29 Statement of General Joseph W. Ralston, The Vice Chairman of the Joint Chiefs of Staff, to the Senate Armed Services Committee, 10 
March 1999. 
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To bring the historical context up to date, there are two Major Acquisition Program 
examples in which OR tools were used effectively. These examples may be instructive 
for ISR investment decision making. 

Joint Direct Attack Munition Program 

Operations Research techniques were the key element in the process that led to the 
definition of the requirements and optimization of the trade space for the Joint Direct 
Attack Munition (JDAM). 

Following Desert Storm, a review of lessons learned identified the fact that adverse 
weather or limited visibility of the ground caused by blowing sand, smoke from ground 
fires, fog and cloud cover limited the employment of precision guided weapons. These 
same conditions also limited the delivery accuracy of unguided weapons when delivered 
from medium and high altitudes. 

Data from Desert Storm was analyzed by a team of operators and requirements officers 
from the Tactical Air Command and the Strategic Air Command (later combined into Air 
Combat Command). This process was facilitated by Glenn Kent of the RAND 
Corporation. The team used optimization techniques to identify the needed key 
characteristics of the new weapon and sensitivity analysis to identify the available trade 
space during acquisition. 

The key characteristics included reduced sensitivity to weather, reduced crew workload 
(laser weapons required human-in-the-loop through most of the flight time), launch and 
leave, and reduced cost over existing precision weapons. Air Force Research Laboratory 
personnel then joined the process and various guidance schemes were traded-off 
against the key characteristics before arriving at a guidance scheme which featured GPS- 
aided inertial guidance. 

The end-game set of requirements and technology trades defined an Adverse Weather 
Precision-Guided Weapon which became the JDAM following industry competition. 

Joint Strike Fighter 

Operations Research methodologies were critical to the Joint Strike Fighter (JSF) 
Requirements Process, Technology Development Activity and affordability trades. The 
JSF program was initiated when OSD was forced to cancel three separate fighter 
development activities with very diverse characteristics. During the first phase of the 
program, called the Joint Advanced Strike Technology Program (JASTP), a joint process 
involving service war-fighters and technologists was employed which identified the key 
characteristics of a next-generation strike system for the USAF, USN, USMC and Royal 
Navy. 

Structured analysis of the needs against scenarios defined by the Joint Staff led to a set 
of key characteristics, associated sensitivity analysis, and a Concept of Operations. 


Operations Research Applications for ISR 





Chapter 3: Operations Research -Historical Context_ 


Industry primes then entered the process and began identifying design space around 
the key characteristics. System design attributes were then analyzed against the key 
characteristics and cost and displayed in carpet plots which allowed senior Service and 
OSD decision makers to understand cost as an independent variable. 

This process resulted in the Services coalescing around a set of requirements that 
allowed satisfaction through a "Family of Airplanes" concept that allowed for a high 
degree of commonality and program affordability. 

As the industry prime contractors refined their design concepts, they were evaluated by 
service warfighters and maintainers in simulation activities conducted by the Program 
Office. This simulation used several core tools provided by and maintained by the 
Program Office. This provided configuration control and increased confidence in the 
simulation results. 

As the program entered the System Design and Development (SDD) phase, this 
simulation activity and the carpet plots provided decision makers the tools to continue 
to make system trades for program affordability. 

3.1 General Lessons Learned from Historical Experience 

There are two important lessons to be learned from the historical examples cited above. 
The first lesson is that, for Operations Research to have a positive impact, the analyst 
should be imbedded in or closely connected to the decision maker's organization. In 
1941, in a note prepared by Professor P.M.S. Blackett entitled "Scientists at the 
Operational Level," he stated, "The object of having scientists in close touch with 
operations is to enable operational staffs to obtain scientific advice on those matters 
which are not handled by the Service technical establishments." In addition, 
"Operational staffs provide the scientists with the operational outlook and data. The 
scientists apply scientific methods of analysis to these data, and are thus able to give 
useful advice." 30 

While outstanding work was done by organizations on both sides of the Atlantic during 
World War I, they were never accepted by the Services they were established to assist. 
In the case of the Naval Consulting Board, they failed to have any great effect on the 
conduct of naval operations because they were a civilian organization that reported 
directly to the Secretary of the Navy. Not once throughout the course of the War did 
the Board establish contact with the operational commanders in the field. 31 

The widespread success of operations research in World War II can be attributed to the 
close relationship the analyst had with the warfighter. In most cases, the analyst was 
deployed to support a specific command in a specific theater of operations. Data were 
readily available at the source, and interactions with staff personnel enabled the analyst 


30 Air Ministry, The Origins and Development of Operational Research in the Royal Air Force, Her Majesty's Stationary Office, London, 1963. 

31 Whitmore, W.F., "Edison and Operations Research," Operations Research, Volume 1, Issue 2 (February 1953), pp. 83-85. 
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to personally demonstrate the utility of operations analysis through near-immediate 
results. 

During the 1960s, McNamara's "Whiz Kids" failed to take notice of the successes 
enjoyed by Operations Research groups during World War II as a result of their close 
alignment with the decision maker - an alignment not present with operators - thereby 
alienating the Services. The result was Service "stiff-arming" of the PPBS process and its 
ultimate failure. With the JROC and JWCA processes, control of the analytic process, 
regardless of its rigor, has been returned to the DoD. 

The second lesson to be learned is that Operations Research can be misused if the 
analysis team is not free from bias. The case in point is the deliberate use of analysis to 
support "Bomber" Harris' position to continue area bombing of German cities during 
World War II rather than redirecting the forces at his disposal to conduct precision 
bombing of military targets (airfields, shipyards, U-boat pens, etc.). As early as the 
autumn of 1944, Strategic Bombing Surveys were initiated by both the United States 
and Britain. These surveys revealed that neither civilian morale nor the German 
economy were as severely impacted as the Bomber Command Operational Research 
Section (ORS) claimed. In fact, it is interesting to note that British analysts stated that 
the effects of the London Blitz were not long-lasting. Analyses conducted by the ORS 
were biased because the analysts were influenced by Harris' strong personality and the 
refusal to depart from pre-war concepts derived from the Japanese attack on Shanghai 
in 1931 and the obliteration of Guernica during the Spanish Civil War by the German 
Condor Legion. 
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Chapter 4: Relevance of Operations Research to Intelligence, 
Surveillance, and Reconnaissance 

The Task Force was tasked by the USD(I) to examine how OR can be used to support ISR 
decision making. Although OR may have wider applicability in broader investment 
decision making, the focus on ISR is driven by several factors that make it both relevant 
and challenging. 

A more consistent scientific and quantitative approach to evaluation could assist the 
USD(I) and the Director of National Intelligence (DNI) with many key investment issues. 
Below are a few examples of past and current issues that illustrate the trade-offs that 
can be illuminated by the proper application of OR. 

4.1 Current ISR Issues that can Benefit from the Application of OR 

In the current conflicts, full motion video (FMV) collection has proven to be a very 
valuable intelligence input. However, studies suggest that a large number of UASs, 
bandwidth and exploitation resources would be required to meet all the anticipated 
requirements for FMV. A program known as the Wide-Area Airborne Sensors is 
designed to collect a wider field persistent view of the battle space as a way of capturing 
information. Understanding the cost/benefit of these programs and their optimum mix 
would be very valuable at this juncture. 

With the focus on current operations, resources allocated to science and technology 
(S&T) collection and exploitation have been reduced. S&T data is essential for 
identifying new adversary capabilities and determining approaches to collect against 
them. The lack of S&T data creates vulnerabilities in the United States' ability to 
counter new threats and collect information against adversaries when they are using 
new technologies. The key to making acquisition decisions is understanding the 
appropriate balance between S&T and current collection capabilities to ensure a 
continued ability to adapt and understand new threats. Acquisition delays and failing 
national systems programs require key investment decisions. These decisions weigh 
trading lower risk and lower capability short term solutions against higher risk and 
higher capability longer term solutions. Prudent investment decisions should be based 
on the optimization of factors such as how UASs and manned aircraft can work with 
national systems to provide support to national and military needs over the next 10 
years. 

In the Global War on Terror, information about individuals can be more important than 
information on military equipment. This has given rise to the desire to collect and use 
biometric data on the battlefield. However, assessing the value of this information and 
how it will be used versus the value of other kinds of collection will be important. In 
addition, as investments in biometrics increase, OR should prove helpful in ensuring that 
investments are made in a balanced fashion across the entire intelligence cycle (i.e., 
requirements, tasking, collection, processing, exploitation, dissemination). 
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4.2 Past ISR Investment Issues that have Benefited from the Application of OR 

The requirements process for the Future Imagery Architecture (FIA), the Integrated 
Overhead SIGINT Architecture (IOSA), the recent U-2 / 
Global Hawk trade-off decision, and the aforementioned 
THE SUCCESSFUL USE OF HVT Study employed OR in a significant manner. In all of 

OPERATIONS RESEARCH these cases, there was a clear mandate from decision 

REQUIRES SOUND makers for the use of OR, general consensus in advance on 

ASSUMPTIONS the objective function, a broad base of community 

involvement, and sound methodology. The Task Force 
notes, however, that the use of OR cannot guarantee 
successful program implementation, particularly if technical capabilities for 
implementation assumed to exist during analysis are not present during design and 
development. As always, the successful use of OR requires sound assumptions. 

4.3 Complications in Applying OR to ISR Investment Issues 

ISR programs are complicated by several factors that make investment decision making 
difficult. 

To add value for a decision maker, information has to be collected, processed, exploited 
and disseminated. Typically the programs that operate in each of these stages are 
controlled by different organizations. Thus, in evaluating an investment decision and its 
impacts, the OR analyst must be careful to look across program elements and across 
organizations. In many cases, the funding streams to support a program may include 
both Military Intelligence Program funding and National Intelligence Program funding. 
The first is controlled by the Secretary of Defense under Title 10 authority, the latter by 
the DNI under Title 50 authority. 

Most ISR systems are used to support a broad variety of missions. For example, the 
same UAS may be tasked to support force protection, battlespace awareness, and 
targeting. National systems are often used to support decisions ranging from national 
strategy and policy to striking individual tactical targets. Evaluating the value of a 
system requires looking at its impact on all of the missions it might support. 

Most major ISR systems are complex and expensive, which generally results in both a 
long development time and a long design life. These factors often result in an ISR 
system operating for more than 20 years after the requirements for the system were 
initially developed. During that time period, threats may change substantially and the 
types of uses for which the system was initially designed may become less relevant. To 
make effective investment decisions throughout the life cycle of ISR assets requires a 
broad view of what the future may hold so that the system can be evaluated against all 
potential future uses. 
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4.4 Challenges in Applying OR to ISR Issues 

The proper use of OR to examine ISR investment decisions objectively can address many 
of the complications identified above. However, applying OR to ISR issues has several 
key challenges. The primary challenge is to answer the question: "What is a pound of 
ISR worth?" 32 That is, what is an objective measure of the value of ISR information? 
Unlike lethal weapon systems, ISR does not directly destroy enemy targets. Instead, it 
produces knowledge that informs decision makers as they make a broad range of 
decisions. This issue makes it difficult to agree on the objective function against which 
OR is performed. To evaluate what this knowledge is worth requires some 
understanding of how decisions are made and what contribution ISR provides to those 
decisions. Although there are no universally-accepted measures that have been 
developed to address this issue, there are some approaches that have been used to 
measure the impact of a specific ISR contribution in specific circumstances. 

The most common ISR metric used for national systems evaluation is "User 
Satisfaction." User Satisfaction allows analysts to quantify the impact of an ISR system 
by measuring the fraction of collection requirements it is able to meet. Using this 
metric, trade-off decisions can be made between architectures or systems by assessing 
the amount of "User Satisfaction" they produce versus the cost of the system. 
Unfortunately, this approach is heavily dependent on the validity of the requirements 
provided, and it assesses only the relative value - not the absolute value - of the 
systems being traded. That is, it does not identify how much is enough or the direct 
impact of this information on outcomes. It is also hard to evaluate systems from 
different collection disciplines, since user requirements are generally specified by 
discipline. 

Other OR approaches attempt to characterize the effect of ISR on outcomes by trying to 
quantify the effect of ISR on attrition of enemy forces, a traditional measure of 
effectiveness. There have been many different ways of addressing this challenge, with 
most suffering from strong limitations. Typically the tactical effect of finding, classifying, 
and locating a target can be related to the rate and probability of killing the target. 
These "sensor-to-shooter" models capture some of the tactical impacts of ISR, but do 
not capture the value of situational awareness or the value of ISR in non-combat 
situations. 

Attempts to capture the wider value of ISR are often stymied by the view held by many 
practitioners that ISR is a craft that is too complex to be characterized. This view makes 
it difficult to address quantitative measures for system performance. As a result, the 
community will have to continue to study the impact of ISR on decision making and 
outcomes to better characterize its value. The development of community accepted 
metrics and the agreement by decision makers on objective functions in advance of an 
OR effort is essential to the successful application of OR. 


32 As quoted by the Honorable James R. Clapper, Jr., Under Secretary of Defense for Intelligence. 
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Chapter 5: Current Operations Research Practices (TOR Tasks 1 and 2) 

Given the Task Force's understanding of OR, its attributes, the conditions under which it 
can be employed successfully, and in response to TOR Tasks 1 and 2, the Task Force 
looked at several specific examples of OR from a broad array of organizations across the 
Department of Defense, the Intelligence Community (1C), and industry. Given the time 
available to the Task Force, it was not possible to explore the entirety of OR and OR- 
related activities in the Department and the 1C. However, the representative sampling, 
while not exhaustive, provided the Task Force with a good opportunity to assess the 
extent to which OR is institutionalized, how it is being employed, and how its 
employment might be improved. This approach should allow the USD(I) to infer the 
extent to which OR is being used to support ISR investment decisions. A list of 
organizations that made presentations is located in Appendix C. 

The Task Force received presentations from entities from each of the Services, from the 
National Geospatial-Intelligence Agency (NGA), the National Security Agency (NSA), the 
National Reconnaissance Office (NRO), ODNI, OSD PA&E, and others, including industry. 
Edge Consulting, Inc. provided a briefing on their ongoing work regarding ISR in support 
of the search for High Value Targets. The Navy has pockets of excellence where OR is 
conducted, and most importantly has a strong curriculum for OR at the Naval 
Postgraduate School (NPS). The Army, which has made the most consistent 
commitment to institutionalizing OR, uses the NPS 
to great advantage to educate their OR workforce. 
The Air Force also maintains an OR degree program 
at the Air Force Institute of Technology (AFIT). The 
most robust OR at the OSD level is performed by 
PA&E. Of note, it is now almost exclusively 
outsourced with little internal capability. The 
Intelligence Community brought multiple examples 
to the Task Force of the ongoing decline in OR 
capability in support of planning, programming, and 
materiel acquisition. FedEx presented a compelling example of a model used in the 
private sector for OR institutionalization and application. Finally, the expertise 
embodied in the MORS, a professional organization dedicated to defense-related OR, 
represents a unique resource for the Department. 

From an overarching perspective, there are some findings that became clear to the Task 
Force. While OR is used widely across the Services, especially in support of operations, 
the application of OR is generally uncoordinated, sometimes organizationally biased, 
inconsistently resourced, and not well integrated into processes and programs. There is 
no strong central champion for OR at the Department level, and therefore there is no 
clear ability to identify and support best practice transfer, or to capture, document, or 
share OR lessons learned. Consequently there is not a jointly accepted OR review or 
accrediting organization. While Federal Acquisition Regulations require OR for Major 


...THE APPLICATION OF OR IS 
GENERALLY UNCOORDINATED, 
SOMETIMES ORGANIZATIONALLY 
BIASED, INCONSISTENTLY 
RESOURCED, AND NOT WELL 
INTEGRATED... 
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Acquisition Programs, the application of OR in response to this requirement is 
inconsistent in level of effort and quality, and often requires rework when submitted for 
OSD review. 

Efforts to support and resource OR have generally been in decline in the Department, 
but this trend is even more stark in the Intelligence Community. In the cases of the 
NGA, NSA, and the NRO, it was clear that OR capabilities for programming and 
acquisition decision making are atrophying rapidly. The Task Force did note that in 
contrast, these same agencies use significant and effective OR in support of current 
operations. Notably, there is a nascent analytic capability in the Defense Intelligence 
Operations Coordination Center (DIOCC), and while its parent organization, the National 
Intelligence Coordination Center (NIC-C) was originally intended to have a strong OR 
capability, that effort has not been resourced. Finally, there is no evidence of OR 
leadership or direction from the staff of the Director of National Intelligence, nor any 
significant OR capability noted on that staff. The recent decision to terminate the 
Integrated Concepts Development Office (ICDO) is further evidence of the apparent 
decline of interest in OR at the ODNI. 

Of the many presentations given to the Task Force, two stood out in a positive manner. 
First, the Army has continued to support a historically strong OR program. The Army 
has developed and sustained an officer career field for OR which includes opportunities 
for advanced degrees and promotion. The Army has continued to support institutions 
that promulgate OR best practices, including the Army Training and Doctrine Command 
(TRADOC) Analyses Center (TRAC). The Army has traditionally maintained a strong 
champion on the Army Staff with direct access to the most senior leadership. The Army 
uses OR both in support to current operations, (with deployed OR specialists embedded 
with units in theater) as well as to support strategic planning, programming, materiel 
acquisition, and the generation of requirements. 

Second, a model which may have direct applicability to the Department is the 
application of OR in the private sector, as evidenced by FedEx. FedEx has a strong and 
consistent commitment to OR, and uses OR extensively in its decision making, both for 
daily operations and long-term strategic planning. The FedEx corporate model is a very 
small (approximately 200 people) headquarters overseeing multiple robust independent 
operating units. FedEx has a hybrid model for OR which is both centralized and 
distributed. A small OR group of approximately eight to ten people provides corporate- 
level OR support, and each of the line operations maintains their own OR entities. OR is 
valued as part of the culture at all levels of the organization, but also seen through a 
pragmatic lens. While OR is often used to great advantage, OR entities within FedEx 
have no line authority. The leader of the corporate-level OR organization made it clear 
that in no case would OR be utilized if there was not clear leadership support, and 
articulated that even the most well-executed OR would be of no value if the decision 
maker did not support it. 
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FedEx uses OR at the corporate level to determine which lines of business are 
appropriate, how much capability is required to support each line of business, and what 
FedEx should be prepared to invest in support of those lines of business. The FedEx 
corporate OR organization does not provide decision support at the operational level 
within the separate lines of business, but does directly support decision making for the 
C-level (CEO, CFO, CIO, etc.) corporate leadership, and can assist the C-level leaders with 
OR-related expertise as they review the analytics submitted by the operating units. The 
corporate OR entity also provides some measure of oversight to the practice of OR in 
the operating units to ensure quality and consistency. Within the line operating units, 
OR is used to optimize efficiency, to test alternative operating concepts, to guide 
investment priorities, and to support other 
decisions as management deems necessary. 


THERE IS NO CHAMPION ADVOCATE 
IN THE DEPARTMENT, NO COHERNT 
STRATEGY, AND NO ORGANIZATION 
OR STAFF ELEMENT IDENTIFIED TO 
PROVIDE LEADERSHIP IN THIS 
DOMAIN. 


It was clear to the Task Force that OR is used, in 
some form or another, in every Service and the 
intelligence Combat Support Agencies. It was also 
clear that save for a few "points of light," OR is in 
decline and not universally valued in decision 
making, especially at the strategic level, and with 
respect to investment decisions and systems 
acquisition. There is no champion advocate in the 
Department, no coherent strategy, and no organization or staff element identified to 
provide leadership in this domain. Moreover, while the 1C conducts truly world class OR 
in determining the daily allocation of scarce collection resources against myriad 
increasing requirements, there has been an astounding decay in the ability of these 
same agencies to conduct OR in support of planning, programming, and acquisition. In 
the case of both the Army and FedEx, however, the Task Force identified some 
significant best practices that may have immediate use as models for a way ahead in OR 
to support ISR investment decisions. 
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Chapter 6: ISR Test Case(s) Attributes (TOR Task 3) 

In approaching TOR Task 3, which is to recommend an ISR test case in which OR might 
be used to reach a better decision, the Task Force found it useful to examine the 
attributes that possible test cases should possess. It is unlikely that every possible test 
case will be characterized by every attribute discussed herein, but the Task Force is 
persuaded that successful cases should be characterized by all or most elements 
described in the sections below. 

The Task Force considered several test cases and believes two cases in particular would 
serve the USD(I) in establishing ways to employ OR effectively in support of ISR. These 
cases are: 


• Biometrics technology investment and architecture 

• Balance intelligence cycle investment 

These test cases are discussed in more detail in Appendix D. 

6.1 Importance 

The Task Force considers it critical that potential test cases be viewed by ISR decision 
makers as involving issues of great importance - certainly to the ISR community and 
especially to USD(I) and DNI leadership. If the issue(s) chosen to examine with OR do 
not meet the common sense test of importance, they are unlikely to receive the kind of 
leadership "buy-in" and resource support that will be required for success. While it is 
true that the Task Force believes that OR could significantly contribute to ISR decisions 
for a multitude of operational, architectural, and investment or acquisition issues, we 
believe that it is the latter area (investment/acquisition of significant resources) that will 
provide the most unique benefit to the USD(I) and DNI missions. 

6.2 Leadership Commitment 

Closely related to, and even driven by, the importance attribute described above, is the 
issue of leadership commitment. If any selected test case is to successfully demonstrate 
the value of OR in ISR decision making, USD(I) (and probably DNI) leadership must be 
fully committed to the case and the process. The following elements of leadership 
commitment to the test case must be present and evident: 

1. Leadership must ensure that stakeholders agree to the terms of reference 
and assumptions for the test case. It is critical that the questions to be 
answered and assumptions to be used are clear and properly framed. 

2. The test case sponsor must commit to providing the resources necessary to 
seeing the test case through to completion, and should provide a "time- 
certain" date when the project is to be completed if it is to have value in 
decision making. 
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3. The test case sponsor must commit to using the OR results in support of the 
decision. While it is not realistic to believe that an important ISR investment 
decision will be driven solely by the OR analytical results, it should be clear to 
stake-holders that the results were carefully considered in the decision¬ 
making process. This is particularly true if, for some reason, the ultimate 
decision diverges significantly from the direction suggested by the OR 
analysis, and it would be helpful if the decision maker provided feedback as 
to how the OR analysis was used and the degree to which it was helpful 
and/or could have been more useful. 

6.3 OR Process Credibility 

All decision makers and stake-holders involved with the chosen test case must be 
convinced of the credibility and integrity of the OR process, even if they ultimately might 
disagree with the results. This means that everything used in the test case OR process 
must be credible and transparent - terms of reference, questions to be answered, data 
to be used, OR tools to be applied, OR practitioners to be utilized, and the decision(s) 
intended to be supported by the process. In addition, decision makers and stake¬ 
holders must agree in advance on the objective functions against which the analysis is 
conducted. If the initial test case(s) conducted do not "ooze" credibility, the prospects 
for the institutionalization of OR for making ISR investment decisions are slim. 

6.4 Test Case Scope 

The issue of test case scope is a challenge. The Task Force has already highlighted the 
attribute of importance, but to be important, the test case does not have to be "huge" - 
especially in terms of scope. The broader the scope the more difficult it will be to 
achieve stakeholder "buy-in" on the credibility of the process and its contribution to 
good decision making. In this regard, an optimal test case may be one that employs OR 
to examine the potential impacts of decisions that are already being contemplated or 
nearing "finality," which by definition are likely to meet the "importance" test already 
described. Generic examples of these kinds of "impact" examinations are provided 
below: 

1. Examining the impact of a contemplated ISR investment decision on its 
applicability against a range of threats, current and future. For example, 
assume that a costly investment in additional high altitude/long endurance 
UASs equipped for imagery and full motion video is contemplated primarily 
for GWOT, and specifically for OIF and OEF, and that the coverage factor and 
output of those UASs is credibly quantified and modeled. An OR examination 
of the contemplated investment could be conducted in other scenarios - for 
example a "defense of Taiwan" or "conflict with Iran." In this example, the 
need for the UASs for GWOT may be the overriding requirement, but one 
could definitively analyze and understand in a credible OR process whether 
the same UASs equipped with the same sensors will be applicable to the 
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other scenarios and consider future alternate utility (either positive or 
negative) into the investment decision. 

2. Examining a contemplated ISR investment decision in a specific new 
discipline for its potential impact (specifically cost and capability) on other 
parts of the relevant intelligence cycle. For example, consider the impact of 
a costly investment in biometric data collection capability. OR techniques 
could be used to analyze the capability of the entire ISR system to effectively 
use the contemplated new collection capability and/or understand what 
additional costs would have to be incurred. An obvious case in point would 
be whether or not appropriate investment has been made in the analytical 
resources (specific skills, recruitment and training) and dissemination 
capability needed to handle the volume of new product that would be 
produced by a UAS investment. Additionally, in an area like biometrics, are 
there legal or privacy impediments that make the collection capability more 
difficult or more expensive to direct, task, and use, and have the costs of 
these factors been credibly analyzed and considered? 

3. Examining the impact of new war-fighting capabilities on the ability of the 
ISR system to support them. A notable historic instance of this was the 
impact on the ISR system of fielding large numbers of precision munitions 
since Desert Storm and the continuing impact of fielding "extreme precision" 
systems today. For example, GPS-guided weapons are only as precise as the 
quality of the Digital Terrain Elevation Data (DTED) that is collected and 
analyzed to provide exact target coordinates. The demand for 
extraordinarily detailed and high-quality imagery to support precision 
application of military capability (strike or SOF) continues to grow 
significantly. Can OR be used to understand and plan for new weapons or 
strike capabilities on the horizon that will produce new quality or volume 
demands on the ISR system? One possible example of this opportunity 
would be the extension of precision strike into ground and naval force 
artillery. The dramatic increase in precision strike potential brought on by 
such an exponential increase in precision weapons would potentially bring 
about a similar exponential increase in the need for highly accurate target 
geographic coordinates and/or high quality target imagery. OR techniques 
seem ideally suited to examine this critical question. 

4. Examining different intelligence disciplines/portions of the intelligence 
cycle to determine where - for a given problem - investment of limited 
resources can provide the most effective and efficient prospects for mission 
satisfaction. Since this type of examination is often hindered by the fact that 
resource sponsors or political proponents are aligned parochially by 
intelligence discipline, organizational interests, budget authorities, or 
constituent concerns versus mission optimization; unbiased OR using good 
quality data and credible analytical techniques could make a significant 
contribution. Even if the challenges were such that OR did not "drive the 
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decision/' it is still possible that applying good OR techniques could improve 
the quality of the debate and create the environment for a more favorable 
outcome. Examples of the kind of problems are numerous: HUMINT vs. 
SIGINT as the route toward discovering enemy intentions, faster/broader 
dissemination vs. more imagery collection, and/or the value of more 
investment in analysis vs. just adding collection capability. 

6.5 Test Case Timing and Duration 

While not as critical as the other attributes, the Task Force considers it optimal if the 
first test case(s) could begin soon, address a near term issue, and be completed 
relatively quickly. It will obviously create more 1C interest if the OR process dealt with 
issue(s) that are currently important and relevant to ongoing resource or operational 
deliberations. Similarly, if the first test case(s) were suitable for a credible OR analysis 
that could be completed in three to six months it would be helpful in getting an early 
determination of the value of the OR process for ISR decision support. 

6.6 A Note on ISR Test Case Challenges 

While the Task Force believes that the ISR test case attributes cited above are valid, it is 
under no illusion about the difficulties of applying OR in the ISR community. OR is hard 
for ISR and challenges abound: 

1. Typically, models exist primarily for collection systems, making the "SR" 
(surveillance and reconnaissance) part of the ISR acronym much easier to 
model and measure than the "I" (Intelligence). In fact, probably the single 
biggest challenge for using OR for ISR investment decision support is the fact 
that we have never been able to successfully measure or model the value of 
the product of the intelligence process - knowledge about an enemy's 
capabilities and intent. In short, the impact of intelligence on decision 
making is not easily measured or understood. 

2. ISR platforms are often multi-purpose, making their total output and relative 
value difficult to measure, model, and compare. 

3. The ISR community lacks analytic models that reflect the utility of ISR product 
assessment. 

4. Funding and control streams are in different organizations, and often in 
different departments of government. 

5. The same ISR platforms and agencies often serve many different customers 
whose requirements range from tactical through strategic, and whose 
missions can range from combat to operational planning or national level 
policy. 
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Chapter 7: Institutionalizing Operations Research (TOR Task 4) 

The Task Force looked at a number of current OR and decision-support institutions in 
use by the Services, DoD, the 1C, and industry. The Task Force found that OR, systems 
analysis, and other decision-support disciplines (including modeling and simulation) are 
used widely in the Services and DoD, as well as the 1C. However, the application of OR 
(and related disciplines) is uneven in these organizations, as is the extent to which OR is 
recognized as a distinct discipline and career field. Some organizations (e.g., the NRO) 
use OR to optimize the use of current ISR resources to support specific mission 
requirements, while others (e.g., the NSA) employ modeling and simulation to support 
architecture analysis. The former example represents robust and sustained use of OR, 
albeit without strong coupling of the resources of OR analysis to strategic investment 
decision making. The latter example typifies a situation in which structured decision 
making has fallen into disuse and in which its practitioners struggle to find opportunities 
to apply their discipline. Of note, the Army has done the most to sustain an 
institutionalized OR capability, one that is well instantiated at the TRAC. TRAC's Website 
states: 


"TRAC conducts operations research (OR) on a wide range of military 
topics, some contemporary but most often set 5 to 15 years in the future. 

How should Army units be organized? What new systems should be 
procured? How should soldiers and commanders be trained? What are 
the costs and benefits of competing options? What are the potential risks 
and rewards of a planned military course of action? TRAC directly 
supports the mission of the Army's major command, the Training and 
Doctrine Command (TRADOC), to develop future concepts and 
requirements while also serving the decision needs of many military 
clients. TRAC analysis enables change." 33 

While the Task Force did not assess in detail the extent to which TRAC is performing 
successfully in all of the roles described above, our discussions allowed us to conclude 
that TRAC is comprised of dedicated professionals who enjoy leadership commitment 
and support and whose work is visible and highly regarded in the Army planning and 
operational communities. 

As described in Chapter 5, the FedEx example also represents a seemingly robust 
approach to institutionalizing OR and structured decision making. This model employs 
OR at a corporate level to help decide on the lines of business in which the company 
should compete, and at what level. FedEx also employs OR at the line-of-business level 
to support investment decision making and to configure operational resources; it 
provides corporate oversight to ensure the presence of consistent OR standard 
processes and methodology throughout the FedEx family of businesses. The company 


33 http://www.trac.armv.mil/ 
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does not "require" the use of OR so much as the leadership commitment of the 
company to OR-based decision making gives its practitioners substantial influence as 
high value decisions are made. That leadership extends to empowering the company's 
corporate OR organizational to ensure that appropriate professional standards, 
processes, and practices are employed in the conduct of OR throughout the company's 
various lines-of-business. 

Four common elements unite the Army and FedEx cases. 

• The OR professionals in both organizations enjoy leadership commitment 
and support. In both organizations, senior leaders employ the results of OR 
in a visible fashion to support key strategic decisions. In the FedEx case, 
discussions regarding important business decisions are expected to be 
supported by corporate or line-of-business specific OR analyses. 
Recommendations not supported risk being dismissed. 

• In both the Army and FedEx, OR is regarded as a discreet career field. OR 
professionals are recruited, and in the case of the Army, developed. OR 
professionals in both organizations enjoy respect; career prospects; and the 
opportunity to develop, refine, and demonstrate their capabilities. In the 
other Services, the Task Force found that a career in Operations Research can 
lead to careers removed from the mainstream resulting in diminished career 
prospects. 

• The Army and FedEx use OR broadly. The TRAC mission unites investment 
decision making with operational concepts. While many companies employ 
OR in decision making, the Task Force found the FedEx model very relevant 
to the functioning of DoD. FedEx uses OR to help decide in which businesses 
to compete and how to allocate resources within specific lines of business in 
support of current operations. In the FedEx process model, OR proves its 
value constantly. 

• The Army and FedEx appear to have aligned closely their respective OR 
cadres, decision makers, and operators. Analyses employing OR appear to 
reflect current operational experience. The Task Force notes that this factor 
is also demonstrated in the historical context in which OR has been used. 

In the past DoD tried to institutionalize structured ISR decision support. In the mid- 
1990's it established a Joint C4ISR Decision Support Center (the Center). In particular, 
the FY97 Defense Planning Guidance (DPG) directed that the Office of the Assistant 
Secretary of Defense for Command, Control, Communications and Intelligence (OASD 
(C3I)) lead an effort in coordination with the USD(AT&L), Joint Staff, CINCS, Services and 
Agencies to develop a DoD Joint C4ISR Conceptual Architecture which would identify 
opportunities to invest in Joint and integrated C4ISR Systems. The purpose of this 
directive was to increase efficiency in the performance of C4ISR systems in Joint 
operations and to reduce the apparent duplicative expenditures in C4ISR systems across 
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the Department of Defense. $10M per year was made available for FY97-2001 in 
support of this effort. The Center's challenge was specific; it was expected to provide 
analysis results to the Joint C4ISR DSC Senior Steering Group, Joint Requirements 
Oversight Council (JROC) and the Defense Acquisition Board (DAB) to assist senior 
decision makers in making Joint C4ISR investment decisions. 

The Center was used actively in support of a number of studies including Sensor-to- 
Shooter, Precision Force C4I Operational Architecture, and other efforts. The Center's 
efforts, however, were not generally coupled to acquisition strategies. Over time, the 
Center's resources diminished and, while it still exists, the Center conducts fewer 
studies and enjoys limited visibility. Overall, the Center lacked sustained leadership 
support, resources, and the mandate to conduct meaningful analyses coupled to 
acquisition and investment decisions. 

The Task Force recognizes that several organizational models are available to the USD(I) 
to institutionalize Operations Research (and scientific method-based decision-support) 
that can address these problems. The Task Force urges the USD(I) and the Department 
to look closely at the private sector and the FedEx model, one in which there exists 
"corporate" governance regarding the necessity to employ OR for key decisions, 
enforcement of OR standards processes and practices, oversight of line-of-business- 
specific OR capabilities, and evaluation of the quality of OR products. The company's 
approach encompasses a compact, but influential corporate organization (for 
governance, oversight, and OR analysis in support of corporate strategy decisions), and 
more fully elaborated OR organizations embedded in the lines of business. The Army's 
TRAC organization may represent a sound example of such an embedded organization, 
performing a wide range of decision-support analyses. However the USD(I) proceeds, 
the MORS represents a powerful domain expert resource that can help build a plan to 
institutionalize OR. 
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Chapter 8: Findings and Recommendations 

The USD(I) should recognize explicitly that the value, quality, and use of OR in support of 
a given decision is dependent on the decision maker's perception of its value and 
relevance; that the quality of ISR decisions can be improved by increased use of high 
quality OR in decision making; and that the state of OR is advancing. 

OR often gets attention on an ad hoc basis, but when immediate demand subsides, 
interest in sustaining its use tends to wane. Institutionalization of OR is missing in the 
overall decision-making process. It is important to develop a set of required standard 
processes and practices, milestones, methodologies, and thresholds for objective 
application of OR to high-value decisions. The Task Force recommends the USD(I) 
closely examine the FedEx and TRAC models. Buttressing these views are four 
important observations: 

1. Operations Research represents a powerful tool to help improve the quality 
of investment decision making by illuminating key issues, assumptions, and 
sources of information. 

2. Operations Research is applied inconsistently throughout the Defense and 
ISR communities. These communities do not possess standard processes and 
practices relating to OR practice, a consistent organizational model, or a 
consistent commitment to the use of OR. 

3. OR - and its use - can be strengthened in the Defense and ISR communities 
through effective institutionalization. 

4. The utility of OR can be more firmly established through appropriate test 
cases. 

The USD(I) should employ OR to support ISR investment decisions relating to the 
dynamic set of capabilities found throughout the intelligence cycle that need to be 
maintained, upgraded, developed, or acquired to satisfy the changing requirements that 
the intelligence cycle must support. This OR capability should be organic to the Office of 
the USD(I), enjoy the full confidence and support of the USD(I), and be comprised of OR 
professionals who can provide objective OR-based recommendations to guide ISR 
investment decisions. 

The Task Force offers the following recommendations: 

1. As Battlespace Awareness portfolio manager, USD(I) should: 

1.1. Establish an organic ISR OR oversight function that requires, sustains, and 
reviews OR standard processes and practices in DoD ISR investment decisions. 

• Consider the approach employed by FedEx as an institutional OR model, 
i.e., decentralized OR-based decision support performed in lines of 
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business; a compact corporate OR organization provides standard 
processes and practices, and oversight of OR use, and conducts OR on 
strategic corporate issues. 

• Consider MORS as a domain expert resource to assist in ISR OR 
institutionalization. 

1.2. Establish a USD(I) capability to conduct OR for strategic investment decisions 
(includes delegation of specific OR analyses to subordinate ISR organizations). 

• Create a professional OR community of interest and practice across DoD 
and the 1C. 

• Coordinate development and application of OR models for specific ISR 
investment decisions. 

1.3. Define requirements for broad based studies for decision making milestones 
and a joint capability for accrediting the studies, models, and data used and 
develop best practices/models for ISR effects. 

1.4. Ensure OR requirements for MAPs and programs below Milestone Decision 
Authority are understood and met. 

2. This OUSD(I) OR function should: 

2.1. Serve as the implementation mechanism for the institutionalization of ISR OR 
within the DoD. 

2.2. Advocate OR as a required input to ISR decision-making processes. 

2.3. Oversee the application of OR to run test cases and scenarios. 

2.4. Apply OR to validate assumptions. 

2.5. Use OR to test investment decisions for proper balance. 

2.6. Track the state of OR and improve OR tools supporting ISR decisions and 
employ them in the decision-making process. 

3. Test Cases: 


3.1. In response to the TOR, the Task Force recommends two test cases: one 
related to investment decision making in biometrics technologies, and one related 
to the balance in the intelligence cycle. These test cases should be structured to 
illuminate challenges and opportunities to strengthen OR decision making in the ISR 
investment domain. 
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THE UNDER SECRETARY OF DEFENSE 

3010 DEFENSE PENTAGON 
WASHINGTON, DC 20301-3010 


JUL 0 5 2007 


MEMORANDUM FOR CHAIRMAN, DEFENSE SCIENCE BOARD 

SUBJECT: Terms of Reference - Defense Science Board (DSB) Advisory Group on Defense 
Intelligence 

You are directed to form a permanent DSB Advisory Group to provide study support 
and independent advice to the Under Secretary of Defense for Intelligence on a broad range 
of defense intelligence matters. The initial review area will be on Improving Intelligence 
Systems Acquisitions. Additional study areas will be formulated over time to support the 
evolving needs of the defense intelligence enterprise. 

The Advisory Group will report via the full DSB to the interested Federal Official. 
Acquisition related matters will be reported to both USD(AT&L) and USD(I), and 
intelligence related matters to USD(I). 

At least one member of the DSB must be a member of the Advisory Group and any 
sub groups formed under the Advisory Group. 

The Task Force shall have access to all classified information, including special 
access programs, relevant to the performance of its tasks. 

Major General Richard O’Lear, USAF (ret.) will serve as the Chairman of this Task 
Force. Ms. Betty Sapp, OUSD(I), will function as the primary Designated Federal Official. 
USD(I) will coordinate on all proposed Task Force members. 

The Task Force will operate in accordance with the provisions of Public Law 92-463, 
the “Federal Advisory Committee Act” and DoD Directive 5105.4, the “DoD Federal 
Advisory Committee Management Program”. We do not anticipate that this Task Force will 
need to go into any ‘particular matters’ within the meaning of Title 18, United States Code, 
section 208, nor will it cause any member to be placed in the position as a procurement 
official. 

This Task Force will be renewed annually or upon change of the Task Force 
Chairman or Designated Federal Official. 


Kenneth J. Krieg 
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OFFICE OF THE UNDER SECRETARY OF DEFENSE 
SOOO DEFENSE PENTAGON 
WASHINGTON. DC 20301-0000 

JAN 2 9 20no 

,NTEU \mMORANDUM FOR CHAIRMAN. DEFENSE SCIENCE BOARD ADVISORY 
GROUP ON DEFENSE INTELLIGENCE 

SUBJECT: Terms of Reference -Operations Research Applications for Intelligence, 
Surveillance and Reconnaissance (ISR) 


You are requested to perform a DSB Advisory Group on Defense Intelligence 
study to examine the use of operations research (OR) methods to support decision¬ 
making within the Department, with a focus on ISR investment choices. 

The principal objective is to assess the use of OR methods within the Department, 
determine how OR supports Department decision-making for ISR programs and to 
recommend improvements, if needed. 

DoD ISR is a large, complex enterprise managed by the Under Secretary of 
Defense for Intelligence (USD(I)). While primary stakeholders for DoD ISR are the 
Combatant Commands, global ISR requirements come from many sources and the low 
volume, high demand DoD ISR assets are continually budget-constrained. Recent use of 
OR analysis to inform the High Value Target study has received positive feedback, and 
there is a wealth of operational data that could be used to expand this analysis across the 
ISR spectrum of requirements. 

The “OR Methods ISR Study” should proceed in four stages: 

(1) Survey the departments and agencies to determine how much OR is being 
performed; 

(2) Assess how the results of that OR are being used in decision making; 

(3) Based on the findings of “(1)” and "(2)”, recommend a test case for using OR 
methodologies to improve DoD ISR investment decisions; 

(4) Recommend steps the Department can take to “institutionalize” the use of OR 
in DoD decision making in the future. 

The study will be sponsored by the Under Secretary of Defense for Intelligence. 
USD(I) is authorized to act upon the advice and recommendations of the Board 

CAPT Keith May, OUSD(I), will serve as the Executive Secretary. 


o 
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The Task Force will operate in accordance with the provisions of P.L. 92-463, the 
“Federal Advisory Committee Act,” and DoD Directive 5105.4, the “DoD Federal 
Advisory Committee Management program.” It is not anticipated that this Task Force 
will need to go into any “particular matters” within the meaning of Section 208 of Title 
18, United States Code, nor will it cause any member to be placed in the position of 
action as a procurement official. 



Deputy Under Secretary of Defense 
(Acquisition, Resources & Technology) 
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Mr. Josh Kerbel, ODNI Lessons Learned 

Center 

ODNI Perspectives on Lessons Learned 

COL Thomas Cioppa, TRADOC 

OIF Strategic Communications Analysis and 
Assessment 

September 17-18, 2008 


Mr. Scott Thayer, NSA 

Cyber Discussion 
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Appendix D: Test Cases 
D. l Biometrics 

The Secretary of Defense (SecDef) has recently taken steps to organize DoD biometrics 
activities. Department of Defense Directive (DoDD) 8521.01E of February 21, 2008 
"...designates the Director, Defense Research & Engineering (DDR&E) as the Principal 
Staff Assistant (PSA) responsible for oversight of DoD biometrics programs and policy..." 
8521.01E also "...designates the Secretary of the Army as the DoD Executive Agent (EA) 
for DoD biometrics..." Section 5.3 of the Directive assigns to the Under Secretary of 
Defense for Intelligence (USD(I)) a broad range of responsibilities to ensure biometrics 
are used effectively to meet intelligence and counter-intelligence challenges. 34 

Although the Directive clarifies DoD Biometrics responsibilities, questions remain 
regarding the application of biometrics to military operations in general, and to the ISR 
domain in particular. Although the Administration has requested funding from Congress 
for biometrics applications and the Army has established a Biometrics Fusion Center 
(BFC), no clear path exists yet for investment in biometrics. Concepts of operations 
(CONOPS) for application of biometrics attributes are lacking, as are the requirements 
important to making biometrics investment decisions. As a result, the DSB Advisory 
Group on Defense Intelligence regards biometrics as an appropriate test case for the use 
of Operations Research in ISR investment decision making. 

In the Task Force's view, OR can be used in three ways pertinent to biometrics 
investment decision making, each of which would help the Department understand OR's 
potential as a decision-support discipline and lead to concrete results in the short term 
relevant to biometrics. 

First, the Department can employ OR to establish and refine concepts of operations for 
the application of biometrics, end-to-end, in support of military operations in general 
and to meet ISR requirements in particular . OR has been used traditionally by the 
Services to model operational requirements, desired effects, and the application of 
resources to meet those requirements and achieve those effects. OR can be used to 
help refine the understanding of the ways in which the insertion of specific biometrics 
data can support and improve the effectiveness of specific military operations. In this 
regard, OR can also help improve the architecture of military operations that employ 
biometrics. 

Second, the USD(I) can work with the Office of the Director of National Intelligence 
(ODNI) to strengthen the information architecture associated with biometrics 
databases, information sharing, as well as biometrics collection, analysis, and 
dissemination. While laudable work has already been done at the National 
Counterterrorism Center (NCTC), Federal Bureau of Investigation (FBI), and elsewhere in 


34 Department of Defense Directive (DoDD) 8521.01E, dated February 21, 2008. 
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the Intelligence Community to employ biometrics information, a unified information 
architecture for the application of biometric attributes (i.e., biometrics) by the 
Intelligence Community does not exist. Work remains to be done to build capabilities to 
query a wide variety of biometrics databases on a federal basis. Indeed, information 
sharing schemes remain rudimentary, dependent on security restrictions, and hampered 
by a lack of data standards. Nonetheless, investment continues in a wide variety of 
programs associated with counter-terrorism and war-fighter support, many of them 
dependent on the availability and use of biometrics data. In addition, the 1C needs, but 
does not possess, a sound analytical concept for the use of biometric attributes, a fact 
that can also impair the ability of the 1C to define biometrics ISR requirements and an 
effective information architecture that encompasses the use of biometrics. Indeed, the 
Task Force is aware of broad concerns that biometrics collection capabilities have 
outpaced the ability of the Intelligence and Defense Communities to share and employ 
biometrics data, both for analysis and for tactical military operations. As a result, the 
"promise" of biometrics has yet to be fulfilled, even through biometrics collection 
capabilities (e.g., retinal, facial, and voice recognition) and tools to "enroll" and compare 
an individual with biometrics data (e.g., the Biometrics Automated Toolset) have 
continued to develop. 

OR should be assessed as a mechanism for defining intelligence operations including 
tasking, collection, and analysis pertinent to biometrics data, and dissemination of 
intelligence products encompassing a biometrics component. Such an application of OR 
could have several distinct benefits: 

• It can lead to better, Community-wide biometrics information architecture. 

• It can help the Community achieve true biometrics information sharing. 

• It can help the USD(I) and others understand how OR might be employed to 
build information-centric inter-agency and Community-wide ISR capabilities. 

Third, the Secretary of the Army and the BFC can use OR to conduct trade studies on a 
variety of specific biometrics technologies and acquisition strategies, consistent with 
DoD acquisition program management requirements. OR can be used to compare 
Technology Readiness Levels, expected investment levels, and likely operational 
capabilities associated with competing technologies, acquisition and investment 
strategies, and funding arrangements. In this instance, OR would employ biometrics as 
a technology development and acquisition strategy test-bed. Such an approach is 
particularly appropriate, given recent moves to strengthen DoD biometrics oversight, 
enhance the financial resources associated with biometrics investment, and upcoming 
acquisition activities on the part of the Biometrics Fusion Center. In this instance, the 
Department should choose an upcoming procurement activity related to biometrics 
collection or enrollment and employ OR to help define and frame a set of investment 
decisions related to two or more specific technologies and two or more specific 
acquisition strategies. 
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Overall, the three elements described above could help the USD(I) understand how OR 
could improve warfighter operations through the use of biometrics as a component of 
ISR, lead to a stronger analytic concept for the use of biometrics, contribute to the 
development of a biometric information architecture, and strengthen biometrics 
technology and program investment decision making. 

D.2 Investing in a Balanced Intelligence Cycle and Imagery UASs - a Value Proposition 

The Intelligence Community has a history of investing in new, discipline-specific 
capabilities without proper consideration of the impact of decisions on the entire 
intelligence cycle. In the past this resulted in large outlays of resources that, because of 
this imbalance, were either not very useful or were marginalized because of an inability 
to realize full value from the investment. For purposes of this discussion, the Task Force 
defines the "Intelligence Cycle" as follows: Planning and Direction, Collection, 
Exploitation, Analysis, Dissemination, and Feedback (Note: The Task Force recognizes 
that, depending upon the collection system being considered, dissemination may occur 
at multiple places in this cycle, for example between collection and exploitation, again 
between exploitation and all-source analysis, and finally to the end-user). Some historic 
examples of where cycle imbalance caused significant problems or challenges include 
the following: 

1. Inability to adequately disseminate national imagery to tactical units during 
Desert Shield/Desert Storm. This is perhaps the most notable example of where 
a lack of balance in the intelligence cycle made a magnificent collection product 
not available in a timely fashion to warfighters in the volume or time-frame 
needed. In this case an enormous investment in imagery collection capability 
was not matched by the investment needed in dissemination capability or 
technology. The result was air strike target imagery or amphibious landing zone 
imagery was available at national and/or combatant command levels, but not to 
those with the mission of striking the targets or landing on the beaches. In 
addition, even if dissemination had been much more capable, imagery analysts 
were not available in the numbers needed for exploitation of the imagery 
product. The reasons for this imbalance are multiple (failure to consider 
dissemination or analysis requirements by a collection system acquirer, failure to 
properly invest in dissemination capability or technology by military Service 
users) so the purpose here is not to try to establish blame, but to point out that 
an OR approach to examining the dissemination and exploitation issues at the 
time of collection system investment decision would have been helpful. 

2. Inability to adequately provide timely feedback to collection system planning 
and direction, especially when the feedback to planning and direction reaches 
across collection disciplines. This challenge is often referred to as "cross- 
cueing," and it is one that is rarely provided for in the design and procurement of 
a collection system. U.S. war-fighting history is replete with examples of where 
adequate and timely feedback from one collection discipline to another (notably 
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among and between SIGINT, ELINT, and IMINT) might have resulted in high-value 
target location or important intelligence discovery. 

3. Inability to adequately provide specialized analysis to support new collection 
capabilities. Measurement and Signature Intelligence (MASINT) is an area where 
new technical collection capability and capacity may be significantly outpacing 
the ability to realize the full value of new kinds of technical collection. Many 
MASINT products require specialized exploitation and specifically trained and 
experienced analysts to make good use of products. It is a problem for 
intelligence cycle effectiveness and utility if investment in exploitation and 
analysis for MASINT does not match the developing collection capacity and it is 
probably better to re-balance the investment than continue to pour more 
MASINT data into the system. An historical example of where an effective 
MASINT intelligence cycle was achieved through balanced investment in 
planning and direction, collection, exploitation, analysis, dissemination and 
feedback was the U.S. Navy's use of Acoustic Intelligence (ACINT) during the Cold 
War. Over several decades, very large investment by the Navy - in both 
operations and intelligence - resulted in strong technical and tactical ACINT 
collection matched by exploitation of the data and use in all-source intelligence 
analysis to provide cueing information to operational forces and technical 
information to weapons systems developers. Coincidentally, this same highly 
developed (and expensive) system (SOSUS, ASWOCS, FOSICS, and FOSIFS) made 
very good use of cross-cueing techniques between ACINT and 
IMINT/SIGINT/ELINT. U.S. Navy anti-submarine warfare is strongly rooted in a 
history of Operations Research, and the degree to which OR played a part in 
developing the ASW/ACINT system might be a useful case study in its own right. 

Significant discussion is underway in the Intelligence Community regarding the 
appropriate balance of resource investment in the Tasking, Processing, Exploitation, and 
Dissemination (TPED) segments of ISR systems. The Task Force suggests that to look 
only at TPED may be "shooting short of the mark." Instead, a test case using OR 
techniques, tools, and practitioners analyze an ISR resource decision with the goal of 
optimizing allocation of resources in a balanced way across the entire intelligence cycle 
as described in paragraph 1 above. A potential test case would be to look at a planned 
investment in imagery UASs, whose collection potential can surely be examined in 
detail using valid coverage factors and collection capability. However, all the 
additional parts of the intelligence cycle should be analyzed similarly with a view 
toward answering the following questions about other parts of the intelligence cycle 
relevant to the potential UAS collection system investment. 

1. Planning and direction : What is the command and control apparatus and are 
there the right number of sufficiently trained personnel in that system to do 
mission planning relevant to the type of missions envisioned? If not, how many 
are needed and what is the cost associated with recruiting, training, and 
retaining them? Are the planned UASs fully compatible with existing ground 
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station infrastructure and is there sufficient capacity in them to handle the 
envisioned new system? If ground station additions/modifications and 
additional ground station personnel are required, have the costs been fully 
analyzed and included in the planned investment? 

2. Collection : Even assuming that the collection capability of the planned UAS 
system has been correctly measured and modeled, have the costs of future 
potential upgrades and/or alternative collection capabilities been estimated? 

3. Dissemination : Is the contemplated new UAS collection capability fully 
compatible with existing dissemination networks and is there sufficient 
communications capacity to handle the increased volume of collection? 
Additionally, once exploitation has been conducted, is there sufficient secondary 
dissemination capacity to get exploited data into the hands of users? The 
bottom line is to ensure that there is sufficient, credible analysis to answer the 
questions relative to primary and secondary dissemination requirements, and 
that associated cost data are included in the investment planning. 

4. Exploitation : An historic problem associated with imagery collection systems is a 
failure to plan and pay for sufficient imagery exploitation personnel to handle 
the increased data quantity. An OR analysis approach could help answer this 
question, as well as provide cost estimates of what the additional personnel and 
their training would cost so that these costs could be included in the cost of the 
overall procurement plan. 

5. Analysis : There is a difference between exploitation and analysis, and if the 
planned new collection capability is to fully contribute to actual knowledge 
development, the capacity must exist to disseminate and incorporate the 
exploited information into the all-source analysis system. While all-source 
analysis is not a part of the collection system per se, pre-decisional OR analysis 
should be conducted to ensure that the envisioned system is fully compatible 
with all source analysis requirements in all parts of the ISR system that could 
benefit from the information. 

6. Feedback : Often the most overlooked part of the intelligence cycle, any new 
system acquisition should include plans for providing feedback - to the entire ISR 
system if required. Perhaps the most sophisticated form of feedback is "cross- 
cueing," where exploited data from one collection system can immediately cue 
(or be cued by) other collection systems. This type of feedback is critical to an 
effective and efficient ISR system and has rarely been incorporated in system 
acquisition decisions, but should be analyzed and accounted for during 
investment decision deliberations. 
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ACINT Acoustic Intelligence 

AFIT Air Force Institute of Technology 

AOR Area of Responsibility 

ARCIC Army Capabilities Integration Center 

A/S Assistant Secretary 

ASD Assistant Secretary of Defense 

ASW Anti-Submarine Warfare 

ASWORG Anti-Submarine Warfare Operations Research Group 

BA Battlespace Awareness 

BAT Biometrics Automated Toolkit 

BDC Biometrics Fusion Center 

BFC Biometrics Fusion Center 

C3I Command, Control, Communications, and Intelligence 

C4ISR Command, Control, Communications, Computers, Intelligence, 

Surveillance and Reconnaissance 

CAA Center for Army Analysis 

CIA Central Intelligence Agency 

CINCs Commanders in Chief 

CJCS Chairman of the Joint Chiefs of Staff 

COCOM Combatant Command 

CONOPS Concept of Operations 

DAB Defense Acquisition Board 

DARPA Defense Advanced Research Projects Agency 

DDRE Director of Defense Research and Engineering 

DEPSECDEF Deputy Secretary of Defense 

DFAR Defense Federal Acquisition Regulation 

DIA Defense Intelligence Agency 

DIOCC Defense Intelligence Operations Coordination Center 

DNI Director of National Intelligence 
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DoD 

DoDD 

DPG 

DSB 

DTED 

DTRA 

EA 

ELINT 

FAR 

FBI 

FedEx 

FFRDC 

FIA 

FMV 

FYDP 

GWOT 

HUMINT 

HVT 

1C 

ICDO 
I OS A 
ISR 

JASTP 

JCS 

JDAM 

JROC 

JSF 

JWCA 

MAP 

MASINT 

MIT 


Department of Defense 
DoD Directive 
Defense Planning Guidance 
Defense Science Board 
Digital Terrain Elevation Data 
Defense Threat Reduction Agency 
Executive Agent 
Electromagnetic Intelligence 
Federal Acquisition Regulations 
Federal Bureau of Investigation 
Federal Express 

Federally Funded Research and Development Center 

Future Imagery Architecture 

Full Motion Video 

Future Year Defense Plan 

Global War on Terror 

Human Intelligence 

High Value Target 

Intelligence Community 

Integrated Concepts Development Office 

Integrated Overhead SIGINT Architecture 

Intelligence, Surveillance, and Reconnaissance 

Joint Advanced Strike Technology Program 

Joint Chiefs of Staff 

Joint Direct Attack Munition 

Joint Requirements Oversight Council 

Joint Strike Fighter 

Joint Warfighting Capability Assessment 
Major Acquisition Program 
Measurement and Signature Intelligence 
Massachusetts Institute of Technology 


Defense Science Board Advisory Group on Defense Intelligence 



Appendix E: Acronyms & Glossary of Terms 


MOA Memorandum of Agreement 

MORS Military Operations Research Society 

MOU Memorandum of Understanding 

MWORG Mine Warfare Operations Research Group 

NCTC National Counterterrorism Center 

NDRC National Defense Research Committee 

NGA National Geospatial-Intelligence Agency 

NIC-C National Intelligence Coordination Center 

NPS Naval Postgraduate School 

NSA National Security Agency 

NRO National Reconnaissance Office 

OASD Office of the Assistant Secretary of Defense 

ODNI Office of the Director of National Intelligence 

OIF Operation Iraqi Freedom 

OR Operations Research 

ORS Operational Research Section 

OSD Office of the Secretary of Defense 

OSRD Office of Scientific Research and Development 

OUSD(AT&L) Office of the Under Secretary of Defense for Acquisition, Technology 
and Logistics 

OUSD(I) Office of the Under Secretary of Defense for Intelligence 

PA&E Program Analysis and Evaluation 

PPBS Planning, Programming and Budgeting System 

PSA Principal Staff Assistant 

R&D Research and Development 

RAF Royal Air Force 

RDT&E Research, Development, Test and Evaluation 

S&T Science and Technology 

SA Systems Analysis 

SDD System Design and Development 

SecDef Secretary of Defense 
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SIGINT Signals Intelligence 

SIOP Single Integrated Operational Plan 

SOF Special Operations Forces 

SOSUS Sound Surveillance System 

TOR Terms of Reference 

TPED Tasking, Processing, Exploitation, Dissemination 

TRAC TRADOC Analyses Center 

TRADOC U.S. Army Training and Doctrine Command 

TSWG Technical Support Working Group 

UAS Unmanned Aerial System 

USA United States Army 

USAF United States Air Force 

USD(AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics 

USD(I) Under Secretary of Defense for Intelligence 

USMC United States Marine Corps 

USN United States Navy 

U.S. United States 

VCJCS Vice Chairman of the Joint Chiefs of Staff 
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